Edge enhancement for thermal-visible combined images and cameras

ABSTRACT

Systems and methods directed toward combining visible light and infrared images can include processing visible light image data to determine an edge factor value for a plurality of visible light pixels corresponding to the strength of an edge at that location. The edge factor value can be determined using features from the visible light image data and an edge gain input, which may be adjustable by a user. The edge factor values are combined with an edge midscale value to create a first set of modified visible light image data including pixels emphasized based on the strength of the edge in the visible light image. The modified visible light image data is combined with infrared image data to create combined image data having contribution from the infrared image data and the edge factor values from the visible light image data.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 14/837,757, filed Aug. 27, 2015 and titled “EDGEENHANCEMENT FOR THERMAL-VISIBLE COMBINED IMAGES AND CAMERAS.” The entirecontent of this application is incorporated herein by reference.

BACKGROUND

Thermal imaging cameras are used in a variety of situations. Forexample, thermal imaging cameras are often used during maintenanceinspections to thermally inspect equipment. Example equipment mayinclude rotating machinery, electrical panels, or rows of circuitbreakers, among other types of equipment. Thermal inspections can detectequipment hot spots such as overheating machinery or electricalcomponents, helping to ensure timely repair or replacement of theoverheating equipment before a more significant problem develops.

Depending on the configuration of the camera, the thermal imaging cameramay also generate a visible light image of the same object. The cameramay display the infrared image and the visible light image in acoordinated manner, for example, to help an operator interpret thethermal image generated by the thermal imaging camera. Unlike visiblelight images which generally provide good contrast between differentobjects, it is often difficult to recognize and distinguish differentfeatures in a thermal image as compared to the real-world scene. Forthis reason, an operator may rely on a visible light image to helpinterpret and focus the thermal image. For example, overlapping and/orcombining the visible light image and the thermal image can provide someguidance for the operator. However, in some situations, it can still bedifficult to distinguish edges and boundaries of objects in the thermalimage.

SUMMARY

Aspects of the present disclosure are directed toward systems andmethods for combining corresponding visible light and infrared imagedata. In some embodiments, a system includes a memory capable of storingone or more sets of visible light image data and infrared image data. Asystem can include a processor configured to process a first set ofvisible light image data to determine an edge factor value for eachpixel in the first set of visible light image data. The magnitude of theedge factor value may be representative of the strength of an edge at aparticular location. In some examples, the edge factor value is based onthe visible light image data and an edge gain input. The system caninclude a user interface by which the edge gain input can be adjustable.In some embodiments, processing the visible light image data can includeperforming an embossing process. The embossing process can includedetermining a pseudo-luminance value for each of a plurality of pixelsand processing the pseudo-luminance values via a processing kernel todetermine the emboss value for at least one pixel associated with thekernel.

In some examples, the processor can be configured to combine the edgefactor value associated with each pixel in the first set of visiblelight image data with an edge midscale value to create a first set ofmodified visible light image data. Accordingly, in some such examples,the modified VL image data represents the edge midscale value offset orotherwise adjusted by the edge factor value at each pixel. In someexamples, the edge midscale value can be a scalar value (e.g., aluminance value) or a vector (e.g., having R, G, and B components). Theedge midscale value can include gray or any other color. In someexamples, the edge midscale value may be selectable or adjustable viathe user interface.

In some embodiments, the processor can be configured to combine thefirst set of modified visible light image with a first set of infraredimage data that corresponds to the first set of visible light image datato generate a first set of combined image data. The resultingcombination can include infrared image data combined with valuesrepresentative of the strength of visible light images at correspondingpixels. As such, edges present in the visible light image data may beenhanced or otherwise visible in the combined image.

The infrared image data can include a scalar infrared values (e.g.,infrared intensities) or vector values (e.g., having R, G, and Bcomponents) based on a palettization scheme, for example. In someexamples, the palettization scheme of the infrared image data may beselectable via a user interface. In some examples, combining theinfrared and visible light image data can include blending the infraredand visible light image data, for example, using a blending ratio. Insome such embodiments, blending ratio is variable across the combinedimage. For instance, in some examples, the blending ratio at each pixelin the combined image is based on the edge factor value at correspondingvisible light pixels.

In various examples, visible light and infrared image data can beacquired from a variety of sources. For instance, one or both of visiblelight and infrared image data can comprise image data recalled frommemory. Additionally or alternatively, one or both of visible light andinfrared image data can comprise a stream of image data, for example,acquired from one or more camera modules or other sensors. In someexamples, the system can include one or both of visible light andinfrared camera modules for acquiring visible light image data andinfrared image data, respectively.

Embodiments of some such systems enable a variety of user-adjustablefeatures. For example, in some embodiments, a user may adjust at leastone of an edge gain, an edge midscale value, an infrared palettizationscheme, and at least one aspect of combining the visible light andinfrared image data.

In some examples, a processor may perform one or more such functionsaccording to instructions on a non-transitory computer readable medium.Additionally or alternatively, various such components can be enclosedin or otherwise supported by a housing. For example, a thermal imagingcamera may include infrared and visible light camera modules capable ofgenerating infrared and visible light image data, respectively. Thecamera can further include a processor for processing the infrared andvisible light image data to generate a combined image, and a display onwhich to present the combined image. The camera could be used to performthermographic processes with the benefit of edges from the visible lightimage data in the thermographic images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective front view of an example thermal imaging camera.

FIG. 2 is a perspective back view of the example thermal imaging cameraof FIG. 1.

FIG. 3 is a functional block diagram illustrating example components ofthe thermal imaging camera of FIGS. 1 and 2.

FIG. 4 is a high-level schematic diagram illustrating an exemplarytechnique for enhancing edges in an IR image.

FIGS. 5A and 5B are schematic diagrams illustrating exemplary VL imageprocessing techniques used for generating a modified VL image.

FIG. 6A is a schematic diagram illustrating the combining of an edgefactor value with an edge midscale value in RGB color space to determinea modified VL image pixel.

FIG. 6B is a schematic diagram illustrating the combining of an edgefactor value with an edge midscale value in YCbCr color space todetermine a modified VL image pixel.

FIGS. 7A and 7B are schematic diagrams illustrating exemplary techniquesfor combining the modified VL image and a corresponding IR image.

FIGS. 8A and 8B are schematic diagrams illustrating a process forgenerating a blended image.

FIGS. 9A and 9B are a flow diagram showing a series of example imagesused in creating a blended image.

FIG. 10 is a series of images illustrating adjustability of edge gainparameters.

FIG. 11 is a process-flow diagram illustrating an exemplary process forcreating a blended image.

DETAILED DESCRIPTION

The following detailed description is exemplary in nature and is notintended to limit the scope, applicability, or configuration of theinvention in any way. Rather, the following description provides somepractical illustrations for implementing various embodiments of thepresent invention. Examples of constructions, materials, dimensions, andmanufacturing processes are provided for selected elements, and allother elements employ that which is known to those of ordinary skill inthe field of the invention. Those skilled in the art will recognize thatmany of the noted examples have a variety of suitable alternatives.

A thermal imaging camera may be used to detect heat patterns across ascene, including an object or objects, under observation. The thermalimaging camera may detect infrared radiation given off by the scene andconvert the infrared radiation into an infrared image indicative of theheat patterns. In some embodiments, the thermal imaging camera may alsocapture visible light from the scene and convert the visible light intoa visible light image. Depending on the configuration of the thermalimaging camera, the camera may include infrared optics to focus theinfrared radiation on an infrared sensor and visible light optics tofocus the visible light on a visible light sensor.

Various embodiments provide methods and systems for producing thermalimages with reduced noise using averaging techniques. To further improveimage quality and eliminate problems that may arise from averaging (e.g.blurring, ghosting, etc.), an image alignment process is performed onthe thermal images prior to averaging.

FIGS. 1 and 2 show front and back perspective views, respectively of anexample thermal imaging camera 100, which includes a housing 102, aninfrared lens assembly 104, a visible light lens assembly 106, a display108, a laser 110, and a trigger control 112. Housing 102 houses thevarious components of thermal imaging camera 100. The bottom portion ofthermal imaging camera 100 includes a carrying handle 118 for holdingand operating the camera via one hand. Infrared lens assembly 104receives infrared radiation from a scene and focuses the radiation on aninfrared sensor for generating an infrared image of a scene. Visiblelight lens assembly 106 receives visible light from a scene and focusesthe visible light on a visible light sensor for generating a visiblelight image of the same scene. Thermal imaging camera 100 captures thevisible light image and/or the infrared image in response to depressingtrigger control 112. In addition, thermal imaging camera 100 controlsdisplay 108 to display the infrared image and the visible light imagegenerated by the camera, e.g., to help an operator thermally inspect ascene. Thermal imaging camera 100 may also include a focus mechanismcoupled to infrared lens assembly 104 that is configured to move atleast one lens of the infrared lens assembly so as to adjust the focusof an infrared image generated by the thermal imaging camera.Additionally or alternatively, the focus mechanism may move the FPArelative to one or more lenses of the infrared lens assembly.

In operation, thermal imaging camera 100 detects heat patterns in ascene by receiving energy emitted in the infrared-wavelength spectrumfrom the scene and processing the infrared energy to generate a thermalimage. Thermal imaging camera 100 may also generate a visible lightimage of the same scene by receiving energy in the visiblelight-wavelength spectrum and processing the visible light energy togenerate a visible light image. As described in greater detail below,thermal imaging camera 100 may include an infrared camera module that isconfigured to capture an infrared image of the scene and a visible lightcamera module that is configured to capture a visible light image of thesame scene. The infrared camera module may receive infrared radiationprojected through infrared lens assembly 104 and generate therefrominfrared image data. The visible light camera module may receive lightprojected through visible light lens assembly 106 and generate therefromvisible light data.

In some examples, thermal imaging camera 100 collects or captures theinfrared energy and visible light energy substantially simultaneously(e.g., at the same time) so that the visible light image and theinfrared image generated by the camera are of the same scene atsubstantially the same time. In these examples, the infrared imagegenerated by thermal imaging camera 100 is indicative of localizedtemperatures within the scene at a particular period of time while thevisible light image generated by the camera is indicative of the samescene at the same period of time. In other examples, thermal imagingcamera may capture infrared energy and visible light energy from a sceneat different periods of time.

Visible light lens assembly 106 includes at least one lens that focusesvisible light energy on a visible light sensor for generating a visiblelight image. Visible light lens assembly 106 defines a visible lightoptical axis which passes through the center of curvature of the atleast one lens of the assembly. Visible light energy projects through afront of the lens and focuses on an opposite side of the lens. Visiblelight lens assembly 106 can include a single lens or a plurality oflenses (e.g., two, three, or more lenses) arranged in series. Inaddition, visible light lens assembly 106 can have a fixed focus or caninclude a focus adjustment mechanism for changing the focus of thevisible light optics. In examples in which visible light lens assembly106 includes a focus adjustment mechanism, the focus adjustmentmechanism may be a manual adjustment mechanism or an automaticadjustment mechanism.

Infrared lens assembly 104 also includes at least one lens that focusesinfrared energy on an infrared sensor for generating a thermal image.Infrared lens assembly 104 defines an infrared optical axis which passesthrough the center of curvature of lens of the assembly. Duringoperation, infrared energy is directed through the front of the lens andfocused on an opposite side of the lens. Infrared lens assembly 104 caninclude a single lens or a plurality of lenses (e.g., two, three, ormore lenses), which may be arranged in series. In some examples, theinfrared lens assembly 104 may include lenses having diffractive orreflective properties or elements. Additional optical components such asmirrors (e.g., Fresnel mirrors) and the like may be included within orotherwise proximate to the infrared lens assembly 104.

As briefly described above, thermal imaging camera 100 includes a focusmechanism for adjusting the focus of an infrared image captured by thecamera. In the example shown in FIGS. 1 and 2, thermal imaging camera100 includes focus ring 114. Focus ring 114 is operatively coupled(e.g., mechanically and/or electrically coupled) to at least one lens ofinfrared lens assembly 104 and configured to move one or both of the FPAand the at least one lens to various focus positions so as to focus theinfrared image captured by thermal imaging camera 100. Focus ring 114may be manually rotated about at least a portion of housing 102 so as tomove the at least one lens to which the focus ring is operativelycoupled. In some examples, focus ring 114 is also operatively coupled todisplay 108 such that rotation of focus ring 114 causes at least aportion of a visible light image and at least a portion of an infraredimage concurrently displayed on display 108 to move relative to oneanother. In different examples, thermal imaging camera 100 may include amanual focus adjustment mechanism that is implemented in a configurationother than focus ring 114, or may, in other embodiments, simply maintaina fixed focus.

In some examples, thermal imaging camera 100 may include anautomatically adjusting focus mechanism in addition to or in lieu of amanually adjusting focus mechanism. An automatically adjusting focusmechanism may be operatively coupled to at least one lens of infraredlens assembly 104 and configured to automatically move the at least onelens to various focus positions, e.g., in response to instructions fromthermal imaging camera 100. In one application of such an example,thermal imaging camera 100 may use laser 110 to electronically measure adistance between an object in a target scene and the camera, referred toas the distance-to-target. Thermal imaging camera 100 may then controlthe automatically adjusting focus mechanism to move the at least onelens of infrared lens assembly 104 to a focus position that correspondsto the distance-to-target data determined by thermal imaging camera 100.The focus position may correspond to the distance-to-target data in thatthe focus position may be configured to place the object in the targetscene at the determined distance in focus. In some examples, the focusposition set by the automatically adjusting focus mechanism may bemanually overridden by an operator, e.g., by rotating focus ring 114.

During operation of thermal imaging camera 100, an operator may wish toview a thermal image of a scene and/or a visible light image of the samescene generated by the camera. For this reason, thermal imaging camera100 may include a display. In the examples of FIGS. 1 and 2, thermalimaging camera 100 includes display 108, which is located on the back ofhousing 102 opposite infrared lens assembly 104 and visible light lensassembly 106. Display 108 may be configured to display a visible lightimage, an infrared image, and/or a combined image that includes asimultaneous display of the visible light image and the infrared image.In different examples, display 108 may be remote (e.g., separate) frominfrared lens assembly 104 and visible light lens assembly 106 ofthermal imaging camera 100, or display 108 may be in a different spatialarrangement relative to infrared lens assembly 104 and/or visible lightlens assembly 106. Therefore, although display 108 is shown behindinfrared lens assembly 104 and visible light lens assembly 106 in FIG.2, other locations for display 108 are possible.

Thermal imaging camera 100 can include a variety of user input media forcontrolling the operation of the camera and adjusting different settingsof the camera. Example control functions may include adjusting the focusof the infrared and/or visible light optics, opening/closing a shutter,capturing an infrared and/or visible light image, or the like. In theexample of FIGS. 1 and 2, thermal imaging camera 100 includes adepressible trigger control 112 for capturing an infrared and visiblelight image, and buttons 116, which form part of the user interface, forcontrolling other aspects of the operation of the camera. A differentnumber or arrangement of user input media are possible, and it should beappreciated that the disclosure is not limited in this respect. Forexample, thermal imaging camera 100 may include a touch screen display108 which receives user input by depressing different portions of thescreen.

FIG. 3 is a functional block diagram illustrating components of anexample of thermal imaging camera 100. Thermal imaging camera 100includes an IR camera module 200, front end circuitry 202. The IR cameramodule 200 and front end circuitry 202 are sometimes referred to incombination as front end stage or front end components 204 of theinfrared camera 100. Thermal imaging camera 100 may also include avisible light camera module 206, a display 108, a user interface 208,and an output/control device 210.

Infrared camera module 200 may be configured to receive infrared energyemitted by a target scene and to focus the infrared energy on aninfrared sensor for generation of infrared energy data, e.g., that canbe displayed in the form of an infrared image on display 108 and/orstored in memory. Infrared camera module 200 can include any suitablecomponents for performing the functions attributed to the module herein.In the example of FIG. 3, infrared camera module 200 is illustrated asincluding infrared lens assembly 104 and infrared sensor 220. Asdescribed above with respect to FIGS. 1 and 2, infrared lens assembly104 includes at least one lens that takes infrared energy emitted by atarget scene and focuses the infrared energy on infrared sensor 220.Infrared sensor 220 responds to the focused infrared energy bygenerating an electrical signal that can be converted and displayed asan infrared image on display 108.

Infrared sensor 220 may include one or more focal plane arrays (FPA)that generate electrical signals in response to infrared energy receivedthrough infrared lens assembly 104. Each FPA can include a plurality ofinfrared sensor elements including, e.g., bolometers, photon detectors,or other suitable infrared sensor elements. In operation, each sensorelement, which may each be referred to as a sensor pixel, may change anelectrical characteristic (e.g., voltage or resistance) in response toabsorbing infrared energy received from a target scene. In turn, thechange in electrical characteristic can provide an electrical signalthat can be received by a processor 222 and processed into an infraredimage displayed on display 108.

For instance, in examples in which infrared sensor 220 includes aplurality of bolometers, each bolometer may absorb infrared energyfocused through infrared lens assembly 104 and increase in temperaturein response to the absorbed energy. The electrical resistance of eachbolometer may change as the temperature of the bolometer changes. Witheach detector element functioning as a sensor pixel, a two-dimensionalimage or picture representation of the infrared radiation can be furthergenerated by translating the changes in resistance of each detectorelement into a time-multiplexed electrical signal that can be processedfor visualization on a display or storage in memory (e.g., of acomputer). Processor 222 may measure the change in resistance of eachbolometer by applying a current (or voltage) to each bolometer andmeasure the resulting voltage (or current) across the bolometer. Basedon these data, processor 222 can determine the amount of infrared energyemitted by different portions of a target scene and control display 108to display a thermal image of the target scene.

Independent of the specific type of infrared sensor elements included inthe FPA of infrared sensor 220, the FPA array can define any suitablesize and shape. In some examples, infrared sensor 220 includes aplurality of infrared sensor elements arranged in a grid pattern suchas, e.g., an array of sensor elements arranged in vertical columns andhorizontal rows. In various examples, infrared sensor 220 may include anarray of vertical columns by horizontal rows of, e.g., 16×16, 50×50,160×120, 120×160, or 650×480. In other examples, infrared sensor 220 mayinclude a smaller number of vertical columns and horizontal rows (e.g.,1×1), a larger number vertical columns and horizontal rows (e.g.,1000×1000), or a different ratio of columns to rows.

In certain embodiments a Read Out Integrated Circuit (ROIC) isincorporated on the IR sensor 220. The ROIC is used to output signalscorresponding to each of the sensor pixels. Such ROIC is commonlyfabricated as an integrated circuit on a silicon substrate. Theplurality of detector elements may be fabricated on top of the ROIC,wherein their combination provides for the IR sensor 220. In someembodiments, the ROIC can include components discussed elsewhere in thisdisclosure (e.g. an analog-to-digital converter (ADC)) incorporateddirectly onto the FPA circuitry. Such integration of the ROIC, or otherfurther levels of integration not explicitly discussed, should beconsidered within the scope of this disclosure.

As described above, the IR sensor 220 generates a series of electricalsignals corresponding to the infrared radiation received by eachinfrared detector element to represent a thermal image. A “frame” ofthermal image data is generated when the voltage signal from eachinfrared detector element is obtained by scanning all of the rows thatmake up the IR sensor 220. Again, in certain embodiments involvingbolometers as the infrared detector elements, such scanning is done byswitching a corresponding detector element into the system circuit andapplying a bias voltage across such switched-in element. Successiveframes of thermal image data are generated by repeatedly scanning therows of the IR sensor 220, with such frames being produced at a ratesufficient to generate a video representation (e.g. 30 Hz, or 60 Hz) ofthe thermal image data.

The front end circuitry 202 includes circuitry for interfacing with andcontrolling the IR camera module 200. In addition, the front endcircuitry 202 initially processes and transmits collected infrared imagedata to a processor 222 via a connection therebetween. Morespecifically, the signals generated by the IR sensor 220 are initiallyconditioned by the front end circuitry 202 of the thermal imaging camera100. In certain embodiments, as shown, the front end circuitry 202includes a bias generator 224 and a pre-amp/integrator 226. In additionto providing the detector bias, the bias generator 224 can optionallyadd or subtract an average bias current from the total current generatedfor each switched-in detector element. The average bias current can bechanged in order (i) to compensate for deviations to the entire array ofresistances of the detector elements resulting from changes in ambienttemperatures inside the thermal imaging camera 100 and (ii) tocompensate for array-to-array variations in the average detectorelements of the IR sensor 220. Such bias compensation can beautomatically controlled by the thermal imaging camera 100 or software,or can be user controlled via input to the output/control device 210 orprocessor 222. Following provision of the detector bias and optionalsubtraction or addition of the average bias current, the signals can bepassed through a pre-amp/integrator 226. Typically, thepre-amp/integrator 226 is used to condition incoming signals, e.g.,prior to their digitization. As a result, the incoming signals can beadjusted to a form that enables more effective interpretation of thesignals, and in turn, can lead to more effective resolution of thecreated image. Subsequently, the conditioned signals are sent downstreaminto the processor 222 of the thermal imaging camera 100.

In some embodiments, the front end circuitry 202 can include one or moreadditional elements for example, additional sensors 228 or an ADC 230.Additional sensors 228 can include, for example, temperature sensors,visual light sensors (such as a CCD), pressure sensors, magneticsensors, etc. Such sensors can provide additional calibration anddetection information to enhance the functionality of the thermalimaging camera 100. For example, temperature sensors can provide anambient temperature reading near the IR sensor 220 to assist inradiometry calculations. A magnetic sensor, such as a Hall Effectsensor, can be used in combination with a magnet mounted on the lens toprovide lens focus position information. Such information can be usefulfor calculating distances, or determining a parallax offset for use withvisual light scene data gathered from a visual light sensor.

An ADC 230 can provide the same function and operate in substantiallythe same manner as discussed below, however its inclusion in the frontend circuitry 202 may provide certain benefits, for example,digitization of scene and other sensor information prior to transmittalto the processor 222 via the connection therebetween. In someembodiments, the ADC 230 can be integrated into the ROIC, as discussedabove, thereby eliminating the need for a separately mounted andinstalled ADC 230.

In some embodiments, front end components can further include a shutter240. A shutter 240 can be externally or internally located relative tothe lens and operate to open or close the view provided by the IR lensassembly 104. As is known in the art, the shutter 240 can bemechanically positionable, or can be actuated by an electro-mechanicaldevice such as a DC motor or solenoid. Embodiments of the invention mayinclude a calibration or setup software implemented method or settingwhich utilize the shutter 240 to establish appropriate bias levels foreach detector element.

Components described as processors within thermal imaging camera 100,including processor 222, may be implemented as one or more processors,such as one or more microprocessors, digital signal processors (DSPs),application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), programmable logic circuitry, or the like, eitheralone or in any suitable combination. Processor 222 may also includememory that stores program instructions and related data that, whenexecuted by processor 222, cause thermal imaging camera 100 andprocessor 222 to perform the functions attributed to them in thisdisclosure. Memory may include any fixed or removable magnetic, optical,or electrical media, such as RAM, ROM, CD-ROM, hard or floppy magneticdisks, EEPROM, or the like. Memory may also include a removable memoryportion that may be used to provide memory updates or increases inmemory capacities. A removable memory may also allow image data to beeasily transferred to another computing device, or to be removed beforethermal imaging camera 100 is used in another application. Processor 222may also be implemented as a System on Chip that integrates some or allcomponents of a computer or other electronic system into a single chip.These elements manipulate the conditioned scene image data deliveredfrom the front end stages 204 in order to provide output scene data thatcan be displayed or stored for use by the user. Subsequently, theprocessor 222 (processing circuitry) sends the processed data to adisplay 108 or other output/control device 210.

During operation of thermal imaging camera 100, processor 222 cancontrol infrared camera module 200 to generate infrared image data forcreating an infrared image. Processor 222 can generate a digital “frame”of infrared image data. By generating a frame of infrared image data,processor 222 captures an infrared image of a target scene atsubstantially a given point in time. That is, in some examples, aplurality of pixels making up the infrared image may be capturedsimultaneously. In other embodiments, sets of one or more pixels may becaptured serially until each pixel has been captured.

Processor 222 can capture a single infrared image or “snap shot” of atarget scene by measuring the electrical signal of each infrared sensorelement included in the FPA of infrared sensor 220 a single time.Alternatively, processor 222 can capture a plurality of infrared imagesof a target scene by repeatedly measuring the electrical signal of eachinfrared sensor element included in the FPA of infrared sensor 220. Inexamples in which processor 222 repeatedly measures the electricalsignal of each infrared sensor element included in the FPA of infraredsensor 220, processor 222 may generate a dynamic thermal image (e.g., avideo representation) of a target scene. For example, processor 222 maymeasure the electrical signal of each infrared sensor element includedin the FPA at a rate sufficient to generate a video representation ofthermal image data such as, e.g., 30 Hz or 60 Hz. Processor 222 mayperform other operations in capturing an infrared image such assequentially actuating a shutter 240 to open and close an aperture ofinfrared lens assembly 104, or the like.

With each sensor element of infrared sensor 220 functioning as a sensorpixel, processor 222 can generate a two-dimensional image or picturerepresentation of the infrared radiation from a target scene bytranslating changes in an electrical characteristic (e.g., resistance)of each sensor element into a time-multiplexed electrical signal thatcan be processed, e.g., for visualization on display 108 and/or storagein memory. When displayed on a display 108, an infrared image cancomprise a plurality of display pixels. Display pixels can have anydefined relationship with corresponding sensor pixels. In some examples,each sensor pixel corresponds to a display pixel in an imagerepresentation of infrared data. In other examples, a plurality ofsensor pixels may be combined (e.g., averaged) to provide infraredinformation for a single display pixel. In still other examples, asingle sensor pixel may contribute to a plurality of display pixels. Forexample, a value from a single sensor pixel may be replicated at nearbypixels, such as in a simple upsampling procedure. In other examples,neighboring or otherwise nearby pixels may be averaged to create a newpixel value, such as in an interpolation procedure. Becauserelationships between display pixels and sensor pixels are defined withrespect to camera operation, the generic term “pixel” may refer to thesensor pixel, the display pixel, or the data as it is processed from thesensor pixel to the display pixel unless otherwise stated. Processor 222may perform computations to convert raw infrared image data into scenetemperatures (radiometry) including, in some examples, colorscorresponding to the scene temperatures.

Processor 222 may control display 108 to display at least a portion ofan infrared image of a captured target scene. In some examples,processor 222 controls display 108 so that the electrical response ofeach sensor element of infrared sensor 220 is associated with a singlepixel on display 108. In other examples, processor 222 may increase ordecrease the resolution of an infrared image so that there are more orfewer pixels displayed on display 108 than there are sensor elements ininfrared sensor 220. Processor 222 may control display 108 to display anentire infrared image (e.g., all portions of a target scene captured bythermal imaging camera 100) or less than an entire infrared image (e.g.,a lesser port of the entire target scene captured by thermal imagingcamera 100). Processor 222 may perform other image processing functions,as described in greater detail below.

Independent of the specific circuitry, thermal imaging camera 100 may beconfigured to manipulate data representative of a target scene so as toprovide an output that can be displayed, stored, transmitted, orotherwise utilized by a user.

Thermal imaging camera 100 includes visible light camera module 206.Visible light camera modules are generally well known. For examples,various visible light camera modules are included in smartphones andnumerous other devices. In some embodiments, visible light camera module206 may be configured to receive visible light energy from a targetscene and to focus the visible light energy on a visible light sensorfor generation of visible light energy data, e.g., that can be displayedin the form of a visible light image on display 108 and/or stored inmemory. Visible light camera module 206 can include any suitablecomponents for performing the functions attributed to the module herein.In the example of FIG. 3, visible light camera module 206 is illustratedas including visible light lens assembly 106 and visible light sensor242. As described above with respect to FIGS. 1 and 2, visible lightlens assembly 106 includes at least one lens that takes visible lightenergy emitted by a target scene and focuses the visible light energy onvisible light sensor 242. Visible light sensor 242 responds to thefocused energy by generating an electrical signal that can be convertedand displayed as a visible light image on display 108. In some examples,the visible light module 206 is configurable by a user, and can provideoutput, for example, to display 108, in a variety of formats. Visiblelight camera module 206 may include compensation functionality forvarying lighting or other operating conditions or user preferences. Thevisible light camera module may provide a digital output including imagedata, which may include data in a variety of formats (e.g., RGB, CYMK,YCbCr, etc.).

Visible light sensor 242 may include a plurality of visible light sensorelements such as, e.g., CMOS detectors, CCD detectors, PIN diodes,avalanche photo diodes, or the like. The number of visible light sensorelements may be the same as or different than the number of infraredlight sensor elements.

In operation, optical energy received from a target scene may passthrough visible light lens assembly 106 and be focused on visible lightsensor 242. When the optical energy impinges upon the visible lightsensor elements of visible light sensor 242, photons within thephotodetectors may be released and converted into a detection current.Processor 222 can process this detection current to form a visible lightimage of the target scene.

During use of thermal imaging camera 100, processor 222 can controlvisible light camera module 206 to generate visible light data from acaptured target scene for creating a visible light image. The visiblelight data may include luminosity data indicative of the color(s)associated with different portions of the captured target scene and/orthe magnitude of light associated with different portions of thecaptured target scene. Processor 222 can generate a “frame” of visiblelight image data by measuring the response of each visible light sensorelement of thermal imaging camera 100 a single time. By generating aframe of visible light data, processor 222 captures visible light imageof a target scene at a given point in time. Processor 222 may alsorepeatedly measure the response of each visible light sensor element ofthermal imaging camera 100 so as to generate a dynamic thermal image(e.g., a video representation) of a target scene, as described abovewith respect to infrared camera module 200. In some examples, thevisible light camera module 206 may include its own dedicated processoror other circuitry (e.g., ASIC) capable of operating the visible lightcamera module 206. In some such embodiments, the dedicated processor isin communication with processor 222 for providing visible light imagedata (e.g., RGB image data) to processor 222. In alternativeembodiments, a dedicated processor for the visible light camera module206 may be integrated into processor 222.

With each sensor element of visible light camera module 206 functioningas a sensor pixel, processor 222 can generate a two-dimensional image orpicture representation of the visible light from a target scene bytranslating an electrical response of each sensor element into atime-multiplexed electrical signal that can be processed, e.g., forvisualization on display 108 and/or storage in memory.

Processor 222 may control display 108 to display at least a portion of avisible light image of a captured target scene. In some examples,processor 222 controls display 108 so that the electrical response ofeach sensor element of visible light camera module 206 is associatedwith a single pixel on display 108. In other examples, processor 222 mayincrease or decrease the resolution of a visible light image so thatthere are more or fewer pixels displayed on display 108 than there aresensor elements in visible light camera module 206. Processor 222 maycontrol display 108 to display an entire visible light image (e.g., allportions of a target scene captured by thermal imaging camera 100) orless than an entire visible light image (e.g., a lesser port of theentire target scene captured by thermal imaging camera 100).

In some embodiments, one or both of infrared 200 and visible light 206camera modules for acquiring IR and VL image data may be included in animage acquisition module 280. The image acquisition module may be inwired or wireless communication with a processing module 290 thatincludes a processor such as 222. Processing module 290 may receiveimage data from the image acquisition module 280 and perform subsequentprocessing steps as will be described herein. In some examples,processing module 290 may include portable processing devices, such as asmartphone, a tablet, a stand-alone computer such as a laptop or desktopPC, or the like. In some such embodiments, various components of frontend circuitry 202 may be included in the image acquisition module 280,the processing module 290, or both.

In these and other examples, processor 222 may control display 108 toconcurrently display at least a portion of the visible light imagecaptured by thermal imaging camera 100 and at least a portion of theinfrared image captured by thermal imaging camera 100. Such a concurrentdisplay may be useful in that an operator may reference the featuresdisplayed in the visible light image to help understand the featuresconcurrently displayed in the infrared image, as the operator may moreeasily recognize and distinguish different real-world features in thevisible light image than the infrared image. In various examples,processor 222 may control display 108 to display the visible light imageand the infrared image in side-by-side arrangement, in apicture-in-picture arrangement, where one of the images surrounds theother of the images, or any other suitable arrangement where the visiblelight and the infrared image are concurrently displayed.

For example, processor 222 may control display 108 to display thevisible light image and the infrared image in a combined arrangement. Insuch an arrangement, for a pixel or set of pixels in the visible lightimage representative of a portion of the target scene, there exists acorresponding pixel or set of pixels in the infrared image,representative of substantially the same portion of the target scene. Invarious embodiments, the size and/or resolution of the IR and VL imagesneed not be the same. Accordingly, there may exist a set of pixels inone of the IR or VL images that correspond to a single pixel in theother of the IR or VL image, or a set of pixels of a different size.Similarly, there may exist a pixel in one of the VL or IR images thatcorresponds to a set of pixels in the other image. Thus, as used herein,corresponding does not require a one-to-one pixel relationship, but mayinclude mismatched sizes of pixels or groups of pixels. Variouscombination techniques of mismatched sized regions of images may beperformed, such as up- or down-sampling one of the images, or combininga pixel with the average value of a corresponding set of pixels. Otherexamples are known and are within the scope of this disclosure.

Thus, corresponding pixels need not have a direct one-to-onerelationship. Rather, in some embodiments, a single infrared pixel has aplurality of corresponding visible light pixels, or a visible lightpixel has a plurality of corresponding infrared pixels. Additionally oralternatively, in some embodiments, not all visible light pixels havecorresponding infrared pixels, or vice versa. Such embodiments may beindicative of, for example, a picture-in-picture type display aspreviously discussed. Thus, a visible light pixel will not necessarilyhave the same pixel coordinate within the visible light image as does acorresponding infrared pixel. Accordingly, as used herein, correspondingpixels generally refers pixels from any image (e.g., a visible lightimage, an infrared image, a combined image, a display image, etc.)comprising information from substantially the same portion of the targetscene. Such pixels need not have a one-to-one relationship betweenimages and need not have similar coordinate positions within theirrespective images.

Similarly, images having corresponding pixels (i.e., pixelsrepresentative of the same portion of the target scene) can be referredto as corresponding images. Thus, in some such arrangements, thecorresponding visible light image and the infrared image may besuperimposed on top of one another, at corresponding pixels. An operatormay interact with user interface 208 to control the transparency oropaqueness of one or both of the images displayed on display 108. Forexample, the operator may interact with user interface 208 to adjust theinfrared image between being completely transparent and completelyopaque and also adjust the visible light image between being completelytransparent and completely opaque. Such an exemplary combinedarrangement, which may be referred to as an alpha-blended arrangement,may allow an operator to adjust display 108 to display an infrared-onlyimage, a visible light-only image, of any overlapping combination of thetwo images between the extremes of an infrared-only image and a visiblelight-only image. Processor 222 may also combine scene information withother data, such as radiometric data, alarm data, and the like. Ingeneral, an alpha-blended combination of visible light and infraredimages can comprise anywhere from 100 percent infrared and 0 percentvisible light to 0 percent infrared and 100 percent visible light. Insome embodiments, the amount of blending can be adjusted by a user ofthe camera. Thus, in some embodiments, a blended image can be adjustedbetween 100 percent visible light and 100 percent infrared.

Additionally, in some embodiments, the processor 222 can interpret andexecute commands from user interface 208, and/or output/control device210. This can involve processing of various input signals andtransferring those signals to the front end circuitry 202 via aconnection therebetween. Components (e.g. motors, or solenoids)proximate the front end circuitry 202 can be actuated to accomplish thedesired control function. Exemplary control functions can includeadjusting the focus, opening/closing a shutter, triggering sensorreadings, adjusting bias values, etc. Moreover, input signals may beused to alter the processing of the image data that occurs in theprocessor 222.

Processor can further include other components to assist with theprocessing and control of the infrared imaging camera 100. For example,as discussed above, in some embodiments, an ADC can be incorporated intothe processor 222. In such a case, analog signals conditioned by thefront-end stages 204 are not digitized until reaching the processor 222.Moreover, some embodiments can include additional on board memory forstorage of processing command information and scene data, prior totransmission to the display 108 or the output/control device 210.

An operator may interact with thermal imaging camera 100 via userinterface 208, which may include buttons, keys, or another mechanism forreceiving input from a user. The operator may receive output fromthermal imaging camera 100 via display 108. Display 108 may beconfigured to display an infrared-image and/or a visible light image inany acceptable palette, or color scheme, and the palette may vary, e.g.,in response to user control. In some examples, display 108 is configuredto display an infrared image in a monochromatic palette such asgrayscale. In other examples, display 108 is configured to display aninfrared image in a color palette such as, e.g., amber, ironbow,blue-red, or other high contrast color scheme. Combinations of grayscaleand color palette displays are also contemplated. In some examples, thedisplay being configured to display such information may includeprocessing capabilities for generating and presenting such image data.In other examples, being configured to display such information mayinclude the ability to receive image data from other components, such asprocessor 222. For example, processor 222 may generate values (e.g., RGBvalues, grayscale values, or other display options) for each pixel to bedisplayed. Display 108 may receive such information and map each pixelinto a visual display.

While processor 222 can control display 108 to concurrently display atleast a portion of an infrared image and at least a portion of a visiblelight image in any suitable arrangement, a picture-in-picturearrangement may help an operator to easily focus and/or interpret athermal image by displaying a corresponding visible image of the samescene in adjacent alignment.

A power supply (not shown) delivers operating power to the variouscomponents of thermal imaging camera 100 and, in some examples, mayinclude a rechargeable or non-rechargeable battery and a powergeneration circuit.

During operation of thermal imaging camera 100, processor 222 controlsinfrared camera module 200 and visible light camera module 206 with theaid of instructions associated with program information that is storedin memory to generate a visible light image and an infrared image of atarget scene. Processor 222 further controls display 108 to display thevisible light image and/or the infrared image generated by thermalimaging camera 100.

As noted, in some situations, it can be difficult to identify anddifferentiate between real-world (visible) features of the target scenein a thermal image. In addition to supplementing the infrared image withvisible light information, in some embodiments, it can be useful toemphasize visible edges within the target scene. While in some instancesthe detection of visible light edges can be difficult to perform in aninfrared image (e.g., across a uniform thermal scene), known edgedetection methods can be performed on a corresponding visible lightimage of the same target scene. Because of the correspondingrelationship between the infrared image and the visible light image,visible light pixels determined to represent a visible edge in thetarget scene correspond to infrared pixels also representing the visibleedge in the infrared image. It will be appreciated that, as used herein,“edges” need not refer to the physical boundary of an object, but mayrefer to any sufficiently sharp gradient in the visible light image.Examples may include physical boundaries of an object, color changeswithin an object, shadows across a scene, and the like.

FIG. 4 is a high-level schematic diagram illustrating an exemplarytechnique for enhancing edges in an IR image. In the illustratedembodiment, a visible light image 400 is processed 402, which mayinclude a variety of processing steps. For instance, in some examples,processing may include filtering the image for noise reduction,enhancing edges, or any other processing techniques. In some examples,processing may include an embossing or other edge detection processesfor emphasizing edges from the VL image.

In the illustrated embodiment, the resulting processed VL image iscombined with an edge midscale value 404 to generate a modified VL image406. The edge midscale value 404 may include, for instance, a specificcolor. In some embodiments, the edge midscale value 404 may beselectable by a user. In some examples, combining the processed VL image402 with the edge midscale value 404 comprises combining the processedVL image with an image consisting exclusively of the edge midscale value404. That is, in some embodiments, a processed VL image is created andan edge midscale value “image” is created or recalled from memory. Thesetwo images are combined by any appropriate combination method togenerate the modified VL image 506.

In other embodiments, the combining can include combining each pixel ofthe processed VL image 402 with the edge midscale value. That is, thecombination is not done image-wide at once, but rather, on apixel-by-pixel or cluster-by-cluster basis. The camera need not store orcreate an edge midscale value “image”, but rather may perform pixel-wisecombinations of processed VL image data and the edge midscale value todetermine modified VL pixel data. The modified VL image 406 may includeemphasized edge information as will be described below. The modified VLimage 406 can be combined with an IR image 408 corresponding to the VLimage 400 to create a blended final image 410. The resulting blendedfinal image 410 may include both IR image data and emphasized edgeinformation from the VL image 400. As described herein, combining (e.g.,the processed VL image 402 and the edge midscale value 404, or the IRimage 408 and the modified VL image 406) may be performed in any of avariety of known image combination techniques, such as addition,averaging, alpha-blending, and the like.

FIG. 5A is a schematic diagram illustrating an exemplary VL imageprocessing technique used for generating a modified VL image. Accordingto the illustrated embodiments, VL image data 500 is directed to a VLprocessing engine 502. In some embodiments, processing engine 502 can bepart of processor 222 or may function separately from processor 222. Insome examples, the VL processing engine 502 can determine apseudo-luminance value 512 for one or more pixels in the VL image. Thepseudo-luminance value 512 of a pixel can include the true luminance(i.e., the Y component in the YCbCr representation of the pixel) or anyother single-valued scalar representation of the pixel. For instance, insome examples, the pseudo-luminance value 512 for a pixel includes alinear combination of the R, G, and B components of the pixelrepresentation in the RGB color space. In an exemplary embodiment, thepseudo-luminance value for a given pixel, i, can be determined by theequation:PseudoLum(i)=4×R(i)+8×G(i)+4×B(i)

Once a pseudo-luminance value 512 is determined for one or more pixels,a processing kernel 514 may be applied across the VL image, for example,across one or more pixels in the VL image 500. In some examples, theprocessing kernel 514 can be used to determine an edge factor value 516for one or more pixels within the kernel that is generallyrepresentative of the “strength” of an edge at a given pixel. In someexamples, determining the edge factor value 516 may include performingan embossing process for determining a value for each pixelrepresentative of the presence and/or strength of a VL edge at thatpixel. With reference to the processing kernel 514 in FIG. 5A, eachentry (1-9) in the kernel 514 may correspond to a single pixel in the VLimage, and each such pixel may have an associated pseudo-luminance value512. In other examples, each entry in the processing kernel 514 maycorrespond to a plurality of pixels within the VL image (e.g., a definedgroup of pixels such as a square or rectangle that may be averagedtogether, such as in a filtering or downsampling process). In someembodiments, the processing kernel 514 as illustrated in FIG. 5A may beused to determine an edge factor value (EFV) 516 for the center pixel ofthe kernel 514. For example, with reference to the illustratedembodiment, the difference in the pseudo-luminance values of diagonallyopposite corner pixels may be used to determine the edge factor value ofthe center pixel. Note that an edge factor value determined by taking adifference between values could be positive or negative. In someexamples, the edge factor value 516 may be scaled by a user-adjustableedge gain 518 value. Thus, in an exemplary embodiment, the edge factorvalue 516 for the center pixel, 5, in the processing kernel 514 can bedetermined by the equation:EFV(5)=edgeGain×[PseudoLum(1)−PseudoLum(9)]

The resulting edge factor value corresponds to the “strength” of an edgeat that pixel. That is, an edge factor value having a large magnitudegenerally corresponds to a sharp edge or contour within the VL image500. It will be appreciated that a variety of equations or methods maybe suitable for determining an edge factor value 516 of one or morepixels in the processing kernel 514. For example, rather thansubtracting associated values for pixels 1 and 9, similar pixelcomparison (e.g., subtraction) may be performed for other pairs orgroups of pixels. In some embodiments, a user may select which pixelsare used for calculating the edge factor value. By doing so, a user mayinfluence how strongly edges in certain directions are reflected in theedge factor values. Similarly, defining which pixels are used indetermining the edge factor value may allow a user to ensure that edgesin a particular direction are less likely to be missed by the EFVcalculation. Exemplary comparisons may include

PseudoLum(3)−PseudoLum(7)

PseudoLum(6)−PseudoLum(4)

PseudoLum(2)−PseudoLum(8)

and the like. Additionally, the kernel need not be limited to a 3×3kernel as shown. Rather, the processing kernel 514 may be any size orshape, such as an n×n square or an m×n rectangle.

In various examples, different equations or methods for determining edgefactor values 516 may be best suited for processing kernels ofparticular size and/or shape. For example, in some embodiments, aplurality of kernel entries may be compared (e.g., via subtraction) andused in determining the EFV. In an exemplary embodiment, a plurality ofsubtractions may be performed (e.g., opposite corners, top and bottomcoordinates, left and right coordinates, etc.) and the subtractionsaveraged for determining the EFT. In another example, the largestdifference magnitude from the set of subtractions is used to determinethe EFV. Additionally or alternatively, in some examples, a singleprocessing kernel 514 may be used to determine edge factor values formultiple pixels. Once the processing engine 502 has determined the edgefactor value 516 for a pixel, the edge factor value 516 may be combinedwith the edge midscale value 504 as described above for generating acorresponding pixel in the modified VL image 506. In general, theprocessing kernel 514 may be moved across the entire VL image in orderto determine an edge factor value for each pixel, or may be moved inorder to determine an edge factor value for a subset of the pixels inthe VL image.

FIG. 5B is a schematic diagram illustrating another exemplary VL imageprocessing technique used for generating a modified VL image. Theprocess of FIG. 5B is similar to that of FIG. 5A, though various stepsmay be performed differently or in different orders. For instance, inthe embodiment of FIG. 5B, the VL image data 500 is processed inprocessing kernel 514. Similarly to the kernel described above withregard to FIG. 5A, kernel 514 of FIG. 5B may be any size, and may beused to process a subset of VL image pixels at a time.

In the illustrated example of FIG. 5B, the processing kernel 514receives the edge midscale value 504 and edge gain 518 parameters. Theprocesses shown as being performed in in FIG. 5A may be performed onpixels in the kernel 514 of FIG. 5B. That is, the pseudo-luminancevalues 512 may be determined for the pixels within the kernel and usedin conjunction with the edge gain 518 to determine an edge factor value516 for one or more pixels within the kernel. The one or more edgefactor values determined within the kernel 514 may be combined with theedge midscale value 504 within the kernel 514.

In some such embodiments, the processing kernel 514 may output one ormore pixels for inclusion in the modified VL image 506. That is, in someembodiments, a subset of VL image pixels are entered into processingkernel 514, in which processing techniques are performed only on thatsubset of pixels. The processing may result in one or more pixels forinclusion in the modified VL image 506. After the processing iscomplete, the kernel 514 may be moved relative to the VL image 500 (or,equivalently, a different subset of pixels of the VL image 500 areapplied to the processing kernel 514) for further processing andgenerating one or more additional pixels for inclusion in the modifiedVL image 506. The process may be repeated for a plurality of subsets ofVL image pixels until the entire modified VL image 506 is constructed.

FIG. 6A is a schematic diagram illustrating the combining of an edgefactor value with an edge midscale value in RGB color space to determinea modified VL image pixel. In the illustrated embodiment, the edgefactor value 616 for a given pixel is combined with each of the R, G,and B values of the edge midscale value 604. In general, the edgemidscale value may be any color. In various embodiments, the edgemidscale value may be a fixed color stored in memory, or may bedefinable or otherwise selectable by a user. Since the edge factor value616 as described above comprises a single value for each pixel, the sameedge factor value 616 may be added to each RGB channel of the edgemidscale value 604, effectively adjusting the R, G, and B values of theedge midscale value 604 to create the modified VL image 606.

In some examples, the edge factor value 616 needs to be scaled to anappropriate size in order to combine the value with each of the RGBchannels. Thus, in the illustrated embodiment, the edge factor value isscaled 620 prior to being added to the various channels. In variousembodiments, the number of bits associated with each channel may not bethe same. For example, in an exemplary embodiment, the R and B channelsmay be 5 bits while the G channel may be 6 bits, using 16 bits total.Any number of bits per channel may be used, and any number of channelsmay have like or different numbers of bits associated therewith.

For a given pixel, adding the EFV 616 to the RGB values of the edgemidscale value 604 defines the resulting color for the correspondingpixel in the modified VL image 606. For example, in the illustratedembodiment, the edge factor value 616 is scaled and added to the Rchannel of the edge midscale value 604 (R_emv), resulting in the Rchannel of the modified VL pixel 606 (R_VL′). The same is performed forthe G and B channels. That is:R_emv+EFV=R_VL′G_emv+EFV=G_VL′B_emv+EFV=B_VL′wherein the edge factor value may be scaled differently for eachchannel.

In general, values in the R, G, and B channels are confined to one ormore certain ranges. For example, in 24-bit color representation, eachof the R, G, and B channels may include 8 bits of information, ranging,for example, between 0 and 255. In other examples, such as a 16-bitcolor representation, each of the R, G, and B channels may be confinedto different ranges. For instance, two channels may be limited to 5 bits(e.g., 0-31) while the third channel is limited to 6 bits (e.g., 0-63).Thus, the sum of an edge midscale value channel and an edge factor valuemay fall outside of the range (e.g., below zero if the edge factor valueis negative or above 255 if the edge factor value is positive). In someexamples, this results in saturation of the channel, and the resultingsum is defined to be the limit of the range. For example, if the Rchannel of the edge midscale value 604 (R_emv) is 240, and the scalededge factor value 616 to be added to R_emv is 50,R_VL′=R_emv+EFV=240+50=295→255That is, even though the sum (295) is above the upper limit of the R_VL′range (255), the resulting value in the modified VL pixel 606 issaturated at the upper limit of the range (255).

Generally, adding the same value (e.g., the edge factor value 616) toeach of the R, G, and B channels of a pixel results in a change in theluminance of the pixel while preserving its chrominance. However, asdescribed above, the edge midscale value 604 may be any color. Invarious embodiments, the edge midscale value 604 may be a specific colorprogrammed into a camera, selectable from a predetermined list, orcustomizable by a user. Thus, in some situations, the R, G, and B valuesof the edge midscale value 604 are not necessarily equal. As a result,adding the edge factor value 616 to each of the R, G, and B channels mayresult in saturation of some channels but not other channels. Forexample, if R_emv=240, G_emv=100, and B_emv=100, and the edge factorvalue 616 is 50, then,R_VL′=240+50=295→255G_VL′=100+50=150B_VL′=100+50=150Thus, in adding the edge factor value 616 to the edge midscale value604, the R channel became saturated, but not the G or B channels. As aresult, the R channel only increased by 15, while the G and B channelsincreased by 50. Saturation of some but not all channels may result in adifference in chrominance between the edge midscale value 604 and themodified VL image 606.

FIG. 6B is a schematic diagram illustrating the combining of an edgefactor value with an edge midscale value in YCbCr color space todetermine a modified VL image pixel. In the illustrated embodiment, theedge factor value 616 is scaled and added to the luminance (Y) channelof the edge midscale value 604 to determine the luminance channel of themodified VL image 606 (Y_VL′). The chrominance channels (Cb, Cr) of theedge midscale value 604 are unaffected by the edge factor value 616, andbecome the chrominance channels of the modified VL image (Cb_VL′ andCr_VL′).

As discussed, adding the edge factor value 616 to the edge midscalevalue 604 only affects the chrominance of the edge midscale value 604 ifone or more, but not all, of the R, G, and B channels of the edgemidscale value 604 become saturated. When this is not the case,chrominance of the edge midscale value 604 is unaffected by the addingof the edge factor value 616, as illustrated in FIG. 6B. Thus, thechrominance channels of the edge midscale value 604 become thechrominance channels of the modified VL image.

In some embodiments, the edge midscale value 604 may be a shade of gray,for example, if R_emv=B_emv=G_emv. It will be appreciated that if suchvalues are scaled to different sizes (e.g., 5-bit vs. 6-bit), the valuesmay not be literally equal, though may represent equivalent valuesrelative to their respective bit depths. Thus, as used herein, being“equal” may refer to numbers that are equivalent in value (e.g.,relative to the number of bits) without necessarily being equal in size(e.g., the number of bits). In such an embodiment, with reference toFIG. 6A, adding the edge factor value 616 to each channel will notresult in saturation of one channel without also saturating the others.That is, each of the R, G, and B channels of the edge midscale value 604are affected uniformly by the edge factor value. Thus, the chrominancechannels of the edge midscale value 604 are unaffected by the edgefactor value, and the representation of FIG. 6B applies.

In an exemplary calculation, if R_emv=G_emv=B_emv, then the edgemidscale value is gray and Cb_emv=Cr_emv. In an exemplary techniqueincorporating 24-bit color depth, Cb_emv=Cr_emv=128. Adding the edgefactor value 616 to each of the R, G, and B channels of the edgemidscale value 604 will not affect the equivalence relationship betweenthe channels, and the chrominance components will be unaffected.Accordingly, after adding the edge factor value 616 to the edge midscalevalue 604, only the luminance value will be affected. That is:Y_VL′=Y_emv+EFVCb_VL′=Cb_emv=128Cr_VL′=Cr_emv=128As shown, since the edge midscale value 604 started out in grayscale(R=G=B; equivalently Cb=Cr=128), the resulting modified VL image 606will also be grayscale. The luminance of the modified VL image 606 willbe the luminance component of the edge midscale value 604 offset (plusor minus) by edge factor value 616.

It will be appreciated that, while FIG. 6B has been discussed in termsof a grayscale edge midscale value 604, in some examples, generating themodified VL image 606 may be performed in the YCbCr color spaceaccording to the embodiment of FIG. 6B whether or not the edge midscalevalue 604 is gray. That is, in some examples, the modified VL image 606may be constructed by adding the edge factor value 616 to the luminancechannel of the edge midscale value 604 and defining the chrominancechannels of the modified VL image 606 to be equal to the chrominancechannels of the edge midscale value 604.

Various combinations of an edge factor value 616 and a generic (e.g.,gray, or any color that may be programmed into memory or selected by theuser) edge midscale value 604 have been described. It will beappreciated that the combination may be performed in a variety of waysin any appropriate color space. RGB and YCbCr color spaces are describedherein by way of example, but similar methods may be performed using,for example, CYMK, HSL, or HSV color space representations. Adding theedge factor value 616 to an edge midscale value 604 may be done in asingle channel (e.g., the luminance channel in FIG. 6B) or a pluralityof channels (e.g., each of the R, G, and B channels in FIG. 6A).

While shown in FIGS. 6A and 6B as being a scalar value added to one ormore channels of the edge midscale value, various other options arepossible. For example, with reference to FIG. 6A, in some embodiments,the edge factor value may be added to fewer than all of the R, G, and Bchannels, such as according to the selection of a user. In doing so,pixels enhanced by large-magnitude edge factor values may be selectivelyshifted toward a certain color in the modified VL image. Similarly, withreference to FIG. 6B, the edge factor value may be added to one or morechrominance channels, in addition to or instead of the luminancechannel, of the edge midscale value.

In some embodiments, the edge factor value may be a multi-channel colorvalue as opposed to a single scalar value. For example, instead ofconverting the VL image to a scalar pseudo-luminance value prior todetermining the edge factor value, the same processing techniques may beperformed on one or more color channels of the VL image. As a result,the edge factor value may comprise a plurality of color channels forcombining with the edge midscale value. Combining of pixels in multiplecolor channels can be performed in a variety of ways (e.g., addition,blending, etc.).

In other embodiments, scalar edge factor values may be determined suchas described above, but scalar values may be mapped to amultiple-channel color space representation. Such a mapping may beperformed in a variety of ways. In some embodiments, scalar edge factorvalues are effectively palettized in a similar way to palettizationschemes of IR image data. The resulting multi-channel edge factor valuesmay be combined with multiple channels of the edge midscale value togenerate the modified VL image. In some such examples, the EFV colorpalette may be chosen to be complementary to the IR palettizationscheme. Examples may include a grayscale IR image with an amber EFVpalette in order to ultimately show visible edges as a degree of amber(yellow/red) within a grayscale IR image. In some embodiments, bothpalettization schemes are selectable by a user. In other embodiments, auser may select one of such palettization schemes and a complementarypalettization scheme is automatically used for the other. In someexamples, while both palettization schemes may be selectable by a user,the user may be alerted via the user interface, after the selection ofone palettization scheme, of possible complimentary palettizationschemes.

The result of the combination is a modified VL image 606 in which eachpixel includes the edge midscale value 604 affected by the edge factorvalue of the corresponding pixel from the VL image. As noted above, themagnitude of the edge factor value 616 generally corresponds to the“strength” of an edge at a corresponding VL pixel. As such, theresulting modified VL image 606 generally includes the edge midscalevalue 604 offset by the edge factor value, and wherein pixelscorresponding to edges in the VL image are most affected by the additionof the edge factor value 616. Accordingly, pixels corresponding to theedges of the VL image are generally the most departed from the edgemidscale value 604 in the modified VL image 606.

With reference to FIG. 4, the modified VL image 406 may be combined witha corresponding IR image 408 to create a blended final image 410. FIGS.7A and 7B are schematic diagrams illustrating exemplary techniques forcombining the modified VL image and a corresponding IR image. As shownin FIG. 7A, the R, G, and B channels of a pixel in the modified VL imageare averaged with the respective R, G, and B channels of a correspondingIR image pixel. The RGB values associated with pixels in the IR image708 may be defined according to a palettization scheme, in which pixelsare assigned to colors within a palette based on associated IR imagedata. As described above, corresponding pixels need not have aone-to-one relationship, but rather can include one or more pixels inthe IR image 708 corresponding to one or more pixels in the modified VLimage 706.

Averaging can include alpha-blending, in which a weighting factordetermines the contribution from each of the IR 708 and modified VLimage 706 pixels. In an exemplary blending procedure, for a blendingratio, alpha (a),R_Blend=α×R_IR+(1−α)×R_VL′G_Blend=α×G_IR+(1−α)×G_VL′B_Blend=α×B_IR+(1−α)×B_VL′

The blending ratio may be user-selectable or adjustable to enable a userto select the amount of blending. As previously described, in someembodiments, alpha can range from 0 to 1 (i.e., 0% IR to 100% IR). Inother embodiments, there may be a minimum and/or maximum blending ratiobetween 0 and 1. In various embodiments the blending ratio may beconstant among all pixels in the image, or may vary from pixel to pixel.In general, the edges from the VL image emphasized by the VL imageprocessing and present in the modified VL image will be accentuated inand effectively enhance the blended image.

In some embodiments, the blending ratio for each pixel may be determinedautomatically based on one or more parameters. In some examples, theblending ratio for each pixel may be a function of the edge factor value(e.g., 616) associated with that pixel. For example, the blending ratiomay vary inversely with the magnitude of the edge factor value. That is,if the magnitude of the edge factor value for a pixel in the VL image issmall, the blending ratio may be large, resulting in a correspondingpixel in the blended image 710 with a relatively large contribution fromthe corresponding IR pixel 708. On the other hand, if the magnitude ofthe edge factor value for a pixel in the VL image is large, the blendingratio may be comparatively small, resulting in a corresponding pixel inthe blended image with a small contribution from the corresponding IRpixel.

The result is a blended image 710 in which pixels not corresponding toedges in the VL image (i.e., a low edge factor value magnitude) have alarge contribution from the IR image. By contrast, pixels correspondingto edges in the VL image (i.e., a large edge factor value magnitude)will have stronger contributions from the modified VL image. As aresult, pixels that do not correspond to edges in the VL image appearsimilar to the corresponding IR pixels from the corresponding IR image.

In an exemplary embodiment, when the edge factor value is below acertain threshold, the blending ratio, α, is set to 1. That is, the whenthere are no edges (or edges below a predetermined or user-adjustablethreshold “strength”) in a pixel in the VL image, the correspondingpixel in the blended image is 100% the corresponding pixel in the IRimage 708. Such pixels retain the RGB values from the original IR image708, and are therefore recognizable on the original RGB palette. Bycontrast, the “stronger” the edge in the VL image, the greater thecontribution of the modified VL image 706 to the blended image 710. Thecolors of the pixels corresponding to “stronger” edges are affected moreby blending than non-edges, and generally do not retain the same RGBvalues from the original IR image. Thus, the edges emphasized by the VLimage processing process can be easily seen in the blended image 710 dueto the blended contribution of the modified VL image 706.

In other embodiments, the blended image 710 will include a minimumamount of IR image contribution and a minimum amount of VL imagecontribution. For example, the blending ratio, α, may be precluded frombeing at least one of 0 and 1, but instead may be limited to a smallerrange of values. It will be appreciated that any limitations on theblending ratio are possible. In addition, other blending schemes may beemployed. For example, the blending ratio may reflect the amount of VLimage contribution to the blended image instead of the amount of IRimage contribution. In some such embodiments, the blending ratio may bea function of the edge factor value, and may increase with increasingedge factor values.

It will be appreciated that, additionally or alternatively, the blendingratio for each pixel may be a function of other parameters, such asparameters of the IR image pixel. For example, in some embodiments, theblending ratio for a given pixel is a function of a magnitude of thepixel in the IR image data (e.g., a scalar value such as a luminance orpseudo-luminance or one or more of associated R, G, or B values in thepalettized IR image). In such examples, various aspects of the thermalscene may contribute to determining the amount of IR image informationand modified VL image information is imported into the blended image. Insome examples, the blending ratio as described above may be inverselyrelated to an IR pixel magnitude, or similarly, the temperature of thetarget scene represented by the IR pixel. That is, in some embodiments,a blended image may include a larger contribution of the modified VLimage at higher temperatures than lower temperatures, generallyemphasizing visible edges in the higher-temperature regions. It will beappreciated that, in general, individual pixel blending ratios dependenton IR intensity may have any functional relationship with the IRintensity values, such as having higher temperature, lower temperature,or a certain range of temperature including a higher or lower modifiedVL image contribution to the blended image.

While a variety of blending techniques have been described, the overallblending ratio may be determined based on a combination of these. Forexample, the overall blending ratio for a given pixel may be acombination of blending ratio determined by parameters such as the edgefactor value and the intensity of corresponding IR pixels. In furtherembodiments, such pixel-dependent blending ratios may also be modifiedby an adjustable global blending radio. In an exemplary embodiment,blending ratios, α_i, are defined per pixel based on the correspondingedge factor value (α_EFV) and the corresponding IR pixel intensity(α_IR), as well as a global blending ratio (α_global). The overallblending ratio for a given pixel may include a combination of suchblending ratios, such as a weighted average:α_net=c_1×α_EFV+c_2×α_IR+c_3×α_globalwherein c1, c2, and c3 are weighting factors. In various embodiments,any parameters involved in the blending process, such as the blendingoptions (e.g., EFV- and/or IR-dependent), values (e.g., global blending)or the relative contribution of each blending technique (e.g., c_i) maybe selectable or adjustable by a user.

It should be noted that, in some embodiments, other factors may bedependent on the IR image intensity in order to selectively emphasizedetails in areas having certain thermal characteristics. For instance,the edge gain value used in determining edge factor values for VL imagepixels may be dependent on the IR image data for corresponding IR imagepixels. That is, VL edges emphasized by the VL image processing may bemore or less emphasized due to the thermal characteristics of acorresponding portion of the target scene. Accordingly, pixels in theblended image having a certain thermal profile (e.g., high temperature,low temperature, within a temperature range, etc.) may include morestrongly emphasized edges than other pixels.

FIG. 7B illustrates an exemplary blending process in YCbCr color space.In the illustrated embodiment, the luminance, Y, and the chrominance, Crand Cr, channels of a modified VL image pixel 706 are averaged withcorresponding channels of a corresponding IR image pixel 708. Asdescribed above, averaging may include weighted averaging using ablending ratio, α. Similar to the RGB blending, in an exemplary blendingprocess,Y_Blend=α×Y_IR+(1−α)×Y_VL′Cb_Blend=α×Cb_IR+(1−α)×Cb_VL′Cr_Blend=α×Cr_IR+(1−α)×Cr_VL′As described above with regard to blending in the RGB color space, for agiven pixel, the blending ratio, α, may be a function of the edge factorvalue associated with the corresponding VL pixel.

As described with regard to FIG. 6B, in some instances, the chrominancecomponents of the modified VL image pixel (Cb_VL′, Cr_VL′) maycorrespond to the chrominance components of the edge midscale value(Cb_emv, Cr_emv). In some such instances, the chrominance components ofa pixel in the blended image 710 may include a weighted blend of thechrominance of the IR image pixel 708 and the chrominance components ofthe edge midscale value. Additionally, in some such instances, theluminance component of the modified VL image pixel (Y_VL′) is theluminance component of the edge midscale value offset by the edge factorvalue associated with the corresponding VL image pixel. Accordingly, theluminance component of a pixel in the resulting blended image 710 mayinclude a weighted blend of the luminance component of the correspondingIR pixel 708 and the luminance component of the edge midscale valueoffset by the edge factor value. It will be appreciated that, while twoexamples are given in FIGS. 7A and 7B, the blending processes such asthose described may be performed in any appropriate color space.

In some examples, none of the images are palettized until the finalblended image. That is, the VL image may be converted into a scalarpseudo-luminance value used to determine a scalar edge factor value. Thescalar edge factor value may be combined with a gray edge midscale valueto create a modified VL image having essentially scalar pixel values.These values may be combined with scalar (grayscale) IR image data togenerate scalar values for pixels in the blended image. A singlepalettization scheme may be applied to the scalar blended image tocreate a colorized image.

In a similar embodiment, pseudo-luminance values may be used todetermine a scalar edge factor value. The scalar edge vector value maybe combined with a gray edge midscale value to create a modified VLimage having essentially scalar pixel values. The resulting modified VLimage may then be palettized according to any appropriate palettizationscheme prior to blending with grayscale or color-palettized IR imagedata.

FIGS. 8A and 8B are schematic diagrams illustrating a process forgenerating a blended image. In FIGS. 8A, R, G, and B channels of a VLimage 800 (R_VL, G_VL, B_VL) are converted into a pseudo-luminance 812(Y′). Pseudo-luminance values 812 are processed to determine an edgefactor value 816 for each pixel, for example, as described above withreference to FIG. 5A. In the illustrated example of FIG. 8A, the edgefactor value 816 is combined with each of the R, G, and B channels(R_emv, G_emv, and B_emv, respectively) of an edge midscale value 804,for example, as described above with reference to FIG. 6A, to generate amodified VL image 806 with channels R_VL′, G_VL′, and B_VL′. Asdescribed above, the combination of the edge factor value 816 with theedge midscale value 804 generally emphasizes pixels corresponding toedges present in the VL image 800.

The R, G, and B channels of the modified VL image 806 (R_VL′, G_VL′,B_VL′) are averaged with corresponding R, G, and B channels of the IRimage 808 (R_IR, G_IR, B_IR) to generate the blended image 810. Blendingcan be performed as described, for example, with reference to FIG. 7A.In some embodiments, the averaging may include a weighted average usinga blending ratio, α, as described above. In some such embodiments, theblending ratio for each pixel may be dependent on the edge factor value816 associated with a corresponding VL pixel, as illustrated by thebroken lines in FIG. 8A. The resulting blended image 810 generallyincludes IR image information and emphasized edges from the contributionof the modified VL image.

FIG. 8B is a schematic diagram illustrating a blending process in whichthe blending is performed in YCbCr color space. The process illustratedin FIG. 8B is similar to that of FIG. 8A, in which the R, G, and Bvalues of the VL image 800 are used to determine a pseudo-luminancevalue 812. It will be appreciated that other methods for determiningpseudo-luminance values using other color space values of the VL image800 may be used. Pseudo-luminance values 812 are used to determine anedge factor value 816 for each pixel. In the illustrated embodiment, asdescribed above with reference to FIG. 6B, the luminance component(Y_emv) of the edge midscale value 804 may be combined with the edgefactor value in order to determine the luminance component of themodified VL image 806. The chrominance components of the edge midscalevalue 804 (Cb_emv, Cr_emv) may be unaffected by the edge factor valueand generally used as the chrominance components of the modified VLimage.

As shown, the Y, Cb, and Cr components of the modified VL image 806 areaveraged with corresponding components of the IR image 808, for example,as described above with reference to FIG. 7B. In such an embodiment, theresulting blended image 810 includes chrominance values that are a blendof the chrominance values of the edge midscale value 804 and the IRimage 808. However, the luminance component of the blended image 810includes the luminance component of the IR image 808 and the luminancecomponent of the edge midscale value 804 offset by the edge factor value816. Thus, edge factor values 816 representative of edges from the VLimage 800 are reflected in the luminance channel of the blended image810 and may therefore be visible in the blended image 810. As describedabove, in some examples, the blending ratio of the IR and modified VLimages at a given pixel may be a function of the edge factor valueassociated with a corresponding pixel in the VL image as illustrated bythe broken lines in FIG. 8B.

In general, as described above with reference to FIG. 5A, the edgefactor value for a given pixel of the VL image may correspond to the“strength” of an edge at that pixel. According to some embodiments asdescribed above, the edge factor value may be used to generate themodified VL image, which is then blended with the IR image. As a result,the “strength” of edges in the VL image is ultimately represented in theblended image. This provides for added clarity and context of items inthe imaged scene in the resulting blended image.

FIG. 9A is a flow diagram showing a series of example images used increating a blended image. As shown, a VL image 900 is processed 902 todetermine edge factor values associated with the VL pixels. In theillustrated embodiment, the VL image 900 is shown in grayscale. In someexamples, the scalar grayscale values of the grayscale pixels may beused as pseudo-luminance values described elsewhere herein. That is, insome such examples, determining a pseudo-luminance value may beimplicitly built in to the VL image.

The edge factor value is then combined with an edge midscale value 904to create a modified VL image 906. As shown, the modified VL image 806appears to be similar to the edge midscale value 904, but darkened orlightened at various locations corresponding to edges seen in the VLimage 900. According to some embodiments, a greater departure from theedge midscale value 904 corresponds to a larger edge factor value (i.e.,a “stronger” edge). Thus, the VL image processing (e.g., determining theedge factor value and combining with the edge midscale value) causesedges within the VL image to stand out from a background that isapproximately the edge midscale value 904 in the modified VL image 906.As described above, in some examples, the edge factor value includes auser-adjustable edge gain parameter, allowing a user to adjust theamount the edges stand out due to the VL image processing.

The resulting modified VL image 906 is subsequently blended with acorresponding IR image 908 to produce a blended image 910. In theillustrated embodiment, the IR image 908 is representative of agenerally uniform thermal scene. That is, there is little temperaturevariation across the scene. As a result, the IR image 908 is generallyconstant across the entire image. However, because of the blending ofthe IR image 908 and the modified VL image 906, the edges emphasized inthe modified VL image 906 are visible in the blended image 910.Additionally, the blended image 910 includes information from the IRimage 908 as well. That is, it is apparent in the blended image 910 thatthe thermal scene is relatively uniform due to little variation outsideof the enhanced edges. In addition, it will be appreciated that whilethe illustrated example demonstrates the IR image being represented by agrayscale palette, any palettization scheme may be used. Similarly,while the VL image is shown in grayscale, other known VL visualizationschemes may be used.

In general, the combination of the IR image data with the modified VLimage data will tend to preserve at least relative temperatureinformation between pixels in the blended image in locations havingsimilar edge factor values (e.g., locations having few or no VL edges).Because the modified VL image data is generally the edge midscale valuemodified by the edge factor value, only those pixels corresponding toedges in the VL image depart significantly from the edge midscale valuein the modified VL image 906. Accordingly, additional detail from the VLimage 900 (e.g., colors or gradual shifts in color or shading, such asdue to variations in scene illumination or light intensity, etc.) doesnot obscure IR image data when creating the blended image 910. Instead,in some embodiments, pixels not corresponding to strong edges in the VLimage (and thus having an associated edge factor value having a lowmagnitude) are generally represented by the edge midscale value 904 inthe modified VL image 906. Such pixels in the modified VL image 906therefore have similar effects on corresponding IR pixels due to theblending process, and the relative relationship between IR pixels withrespect to the original IR palette is generally maintained.

As described above, in some embodiments, the blending ratio forcombining the modified VL image 906 and the IR image 908 at each pixelis a function of the edge factor value associated with the correspondingVL pixel. Thus, in some such embodiments, pixels in the blended image910 not corresponding to strong edges in the VL image 900 may largely orentirely include the IR image data, thereby entirely or nearly entirelypreserving the IR image information at pixels corresponding to no orweak edges in the VL image 900. By contrast, pixels corresponding toedges in the VL image 900 (having a high-magnitude edge factor value)will include the modified VL image data incorporating such edge factorvalues. Thus, the edges in the VL image corresponding to pixels havingedge factor values of a large magnitude will ultimately be incorporatedinto the blended image 910 by way of the edge factor value contributionto the modified VL image 906.

FIG. 9B is another flow diagram showing a series of example images usedin creating a blended image. Generally, the process follows as describedabove with respect to FIG. 9A, in which the VL image 900 is processedand combined with the edge midscale value 904 to generate a modified VLimage 906. The modified VL image 906 includes edges showing the outlineof a person as well as background details (whiteboard and writing). Themodified VL image 906 is combined with the IR image 908 to create ablended image 910. While the thermal scene represented in FIG. 9A isrelatively uniform, the thermal scene represented in FIG. 9B includesthermal variations. That is, the IR image 908 of FIG. 9B includescontents having a wide variety of temperatures (a relatively uniformbackground with higher-temperature person in the foreground). As shown,the blended image 910 of FIG. 9B includes both the thermal informationfrom IR image 908 (the person stands out from the background) as well asvisible details from the VL image (the whiteboard and writing details).Thus, the resulting blended image 910 effectively combines thermalinformation from the IR image data as well as edges and detailinformation from the VL image data.

As previously described, the VL image processing may include anadjustable (e.g., user adjustable) edge gain parameter. Thus, theresulting modified VL image may include an adjustable amount of offsetor other enhancement at pixels corresponding to VL edges. FIG. 10provides a series of modified VL images including adjusted edge gainparameters. As discussed previously, the VL image 1000 may be processedto generate edge factor values for each pixel which may depend on anedge gain parameter. Modified VL images 1006 a, 1006 b, 1006 c, and 1006d are exemplary images resulting from VL image processing havingdifferent edge gain parameters. For example, image 1006 a is createdusing a relatively small edge gain parameter, with low but visiblecontrast present between pixels corresponding to VL edges and those notcorresponding to VL edges. By contrast, image 1006 b is a modified VLimage generated using an edge gain parameter that is larger than that ofimage 1006 a, and the resulting image 1006 b includes increased visiblecontrast at pixels corresponding to VL edges. Similarly, image 1006 c isa modified VL image generated with a still larger edge gain parameterwhen compared to image 1006 b. Accordingly, pixels corresponding to VLedges are further emphasized in image 1006 c when compared to images1006 a and 1006 b. Image 1006 d is a modified VL image generated using astill greater edge gain parameter than image 1006 c, therefore havingeven further offset and/or enhanced pixels corresponding to VL edges.

In various embodiments, a user may adjust the edge gain parameter inorder to affect the degree of enhancement of pixels corresponding to VLedges, for example, via a user interface. In some embodiments, the usermay adjust the edge gain parameter in real time and observe the effectof the adjustment on the display. In some examples, the edge gainparameter may be selectable from a list of predetermined values orpercentages. In other examples, the edge gain parameter may becontinuously adjusted between minimum and maximum values.

FIG. 11 is a process-flow diagram illustrating an exemplary process forcreating a blended image. One or more cameras may be used to acquirecorresponding IR and VL images (1130). The images may be captured by thesame or different cameras, and may be subsequently processed and/oranalyzed on one or both of the cameras, or transferred to an externalprocessing device. The method can include registering the acquired IRand VL images (1132). As previously discussed, one or both of the IR orVL images may be scaled (upsampled, downsampled, etc.) to achieve aone-to-one pixel mapping between the images. Additionally oralternatively, in some examples, a plurality of pixels in one of theimages may correspond to a single pixel or a different-sized pluralityof pixels in the other image. Next, an edge factor value may bedetermined for each pixel in the VL image (1134). The determined edgefactor values from a plurality of pixels can be added to or otherwisecombined with an edge midscale value in order to create a modified VLimage (1136). Finally, the method can include blending the modified VLimage with the IR image in order to generate a final blended image(1138). In some examples, the blending may be performed on apixel-by-pixel basis, with the blending ratio at each pixel being basedon the edge factor value associated with that pixel. As previouslydescribed, in some examples, pixel processing is done on an entire-imagescale, in which edge factor values are combined with an entire “frame”of the edge midscale value. In alternative embodiments, severalprocessing steps may be performed on a subset of pixels within theprocessing kernel. Accordingly, in some embodiments, after performing aseries of processing steps the step of moving the kernel (1150) may beperformed. As shown by the broken lines, in an exemplary process,modified VL image pixels may be blended with corresponding IR pixels(1138) for pixels within the processing kernel. Then the processingkernel may be adjusted (1150) to perform processing steps (e.g.,1134-1138) on a new set of pixels. After generating the blended image(1138), the method can include displaying the blended image (1140), forexample on display 108.

In various examples, the display of the blended image may take a varietyof forms. For instance, in some examples, the entire displayed imageincludes a blended image having a combination of IR image data andmodified VL image data. In other examples, the blended image may bepresented within a larger VL image, for example, in a picture-in-picturedisplay. In general, the blended image may be presented in any wayappropriate for presenting IR image data on a display, such as thosedescribed, for example, in U.S. patent application Ser. No. 12/828,442,filed Jul. 1, 2010, and entitled “THERMOGRAPHY METHODS,” U.S. Pat. No.7,994,480, filed Jul. 19, 2006, and entitled “VISIBLE LIGHT AND IRCOMBINED IMAGE CAMERA,” and U.S. patent application Ser. No. 13/833,853,filed Mar. 15, 2013, and entitled “THERMAL IMAGE ANIMATION,” each ofwhich is assigned to the assignee of the instant application and ishereby incorporated by reference herein in its entirety.

Methods as described herein may be performed in a variety of ways. Insome examples, image processing and blending techniques may be carriedout, for example, by a processor (e.g., 222) in a thermal imagingcamera. In some such examples, one or both of the VL and IR images maybe captured by the same camera or a different camera. In someembodiments, IR and VL images captured by the same or different camerasmay be transferred to an external processing device, such as a computer,tablet, smartphone, or the like, which may perform any number ofprocessing steps (e.g., registration, generating modified VL images,blending, etc.). In some examples, corresponding IR and VL images may becaptured and processed to generate a blended image as described hereinby a single device. In some embodiments, such image capturing andprocessing may be performed in substantially real time, for instance,using a live stream of images to generate a video signal comprising aseries of blended images. In general, as used herein, unlessspecifically stated otherwise, the term “image” may refer to a singleimage frame, such as an image store in and recalled from memory, a stillframe from a video feed, a live or prerecorded video feed comprising aseries of captured still images, a live video feed in which someportions of the display are dynamically updated at different time thanothers, or other known image data presentation schemes.

In some embodiments, one or both of IR and VL image data comprises astream of data received by the processor. In some such embodiments, theprocessor can receive a stream of image data and generate a video filetherefrom. For example, in some embodiments, the processor can receive astream of IR image data and VL image data, and generate therefrom avideo file. In various embodiments, the generated video file cancomprise IR image data, VL image data, or a combination of blended IRand VL image data. In some embodiments, processing techniques describedherein may be performed to the image data stream as it is received bythe processor.

As used herein, “IR” may refer to wavelengths in any portion of theinfrared spectrum, such as LWIR (between approximately 8 microns and 14microns), MWIR (between approximately 3 microns and 5 microns), SWIR(between approximately 1 micron and approximately 2 microns), or anycombination of these ranges or wavelengths therebetween. “VL” imagestypically refer to wavelengths in the visible spectrum (e.g., betweenapproximately 400 nanometers and approximately 700 nanometers). However,the processes as described herein for use with VL images may beperformed using alternative wavelengths, such as NIR (e.g., betweenapproximately 700 nm and 1000 nm) or UV (e.g., between approximately 200nm and 400 nm). In general, processes for combining IR and VL imagesdescribed herein may be performed on any set of two or more images.

Various aspects of methods described herein may be adjusted by a user.For example, a thermal imaging camera capable of executing such methodsmay include a user interface (e.g., 108, 112, 114, 116) for receivingone or more inputs from a user. In some examples, a user may adjust atleast one of, for example, the IR palettization scheme, an edge gainused for determining edge factor values, the edge midscale value, a typeof image blending (e.g., constant across image, EFV-dependent, acombination thereof, and the like), the amount of image blending (e.g.,blending ratios). In other embodiments, one or more of such parametersmay be fixed.

In some embodiments, various edge detection techniques may be includedin the VL image processing, such as described in U.S. patent applicationSer. No. 14/222,153, filed Mar. 21, 2014 and entitled “VISIBLE LIGHTIMAGE WITH EDGE MARKING FOR ENHANCING IR IMAGERY,” which is assigned tothe assignee of the instant application and is hereby incorporated byreference in its entirety. In some such embodiments, pixels determinedto correspond to edges in the VL image may be used in a variety of ways.For instance, in some examples, blending ratios used for each pixel increating the blended image may be dependent on whether or not thecorresponding VL pixel is determined to be an edge pixel. Additionallyor alternatively, edge factor values may be determined for one or moreVL pixels based on whether or not the pixel is determined to be an edgepixel.

Example thermal image cameras and related techniques have beendescribed. The techniques described in this disclosure may also beembodied or encoded in a computer-readable medium, such as anon-transitory computer-readable storage medium containing instructions.Instructions embedded or encoded in a computer-readable storage mediummay cause a programmable processor, or other processor, to perform themethod, e.g., when the instructions are executed. Computer readablestorage media may include random access memory (RAM), read only memory(ROM), a hard disk, optical media, or other computer readable media.

For example, an external computer comprising such computer readablemedium can receive corresponding visible light and infrared images froma thermal imaging camera or from memory and perform edge detectionand/or process the VL and IR images to generate display images asdescribed herein. In some embodiments, various portions of thetechniques can be embodied in multiple components. For example, athermal imaging camera can detect edges in a visible light image andpass detected edge information to an external computing device forgenerating the display image incorporating the detected edges.Additionally or alternatively, the external computing device may assistin or otherwise perform edge detection and/or enhancement techniques.

In further examples, embodiments of the invention can be embodied in adisplay system. The display system can be configured to receive VL andIR image data and carry out processes such as those herein described.Exemplary display systems can include one or more processors, a displayand a user interface for carrying out such processes. A display systemcan be incorporated into any appropriate device or system capable ofreceiving and processing image data. In some embodiments, the displaysystem can include a portable, hand-held thermal imaging camera such asthose described elsewhere herein in order to capture corresponding VLand IR images and provide VL and IR image data to other components ofthe imaging system. In further embodiments, the imaging system is fullyincorporated into such a camera, or can consist essentially of a cameracapable of carrying out any of the various processes described.

Various embodiments have been described. Such examples are non-limiting,and do not define or limit the scope of the invention in any way.Rather, these and other examples are within the scope of the followingclaims.

The invention claimed is:
 1. A thermal imaging system configured togenerate images having infrared image data and enhanced edgescomprising: an infrared camera module for receiving infrared radiationfrom a target scene and generating infrared image data representative ofthe target scene; a visible light camera module for receiving visiblelight radiation from the target scene and generating visible light imagedata representative of the target scene, the visible light image datahaving a plurality of regions corresponding to regions of the targetscene; a processor in communication with the infrared camera module andthe visible light camera modules, the processor configured to: receivevisible light image data from the visible light camera module; receiveinfrared image data corresponding to the received visible light imagedata from the infrared camera module; determine, for each of theplurality of regions in the received visible light image data, acorresponding edge factor value associated with edges in the visiblelight image data; generate modified visible light image data having aplurality of regions corresponding to the plurality of regions in thevisible light image data by combining, for each of the plurality ofregions of the modified visible light image data, an edge midscale valueand the corresponding edge factor value, the edge midscale valuecorresponding to a specific color, and the combining the edge midscalevalue and the corresponding edge factor value comprising, for each ofthe plurality of regions, adjusting a luminance of the edge midscalevalue using the corresponding edge factor value; and combine themodified visible light image data with the corresponding receivedinfrared image data to create combined image data.
 2. The system ofclaim 1, further comprising a user interface in communication with theprocessor, and wherein the processor is configured to receive aselection of the edge midscale value from the user interface.
 3. Thesystem of claim 2, wherein: the user interface comprises a display; theprocessor is configured to present a predetermined list of possible edgemidscale values via the display; and wherein receiving a selection ofthe edge midscale value from the user interface comprises receiving aselection of the edge midscale value from the predetermined list ofpossible edge midscale values.
 4. The system of claim 1, furthercomprising a user interface, and wherein the processor is configured toreceive an edge gain input via the user interface; and determining theedge factor value for each of the plurality of regions of the receivedvisible light image data comprises: performing an embossing process onthe visible light image data in order to determine an emboss value foreach of the plurality of regions within the visible light image data;and scaling the determined emboss value by the edge gain input.
 5. Thesystem of claim 1, further comprising a display in communication withthe processor, and wherein the processor is configured to present thecombined image data as a display image on the display.
 6. The system ofclaim 1, wherein combining the modified visible light image data withthe corresponding received infrared image data comprises, for each ofthe plurality of regions of the modified visible light image data,blending the modified visible light image data with the correspondinginfrared image data, the blending being performed based on a blendingratio α associated with each of the plurality of regions, and wherein,for each of the plurality of regions, the blending ratio α for theregion is based on the edge factor value associated with thecorresponding region in the visible light image data.
 7. The system ofclaim 6, wherein, for each of the plurality of regions of the modifiedvisible light image data, if the edge factor value associated with thecorresponding region in visible light image data is below a thresholdvalue, the blending ratio is set so that the combined image dataincludes a minimum amount of modified visible light image data.
 8. Thesystem of claim 7, wherein the minimum amount of modified visible lightimage data comprises zero percent modified visible light image data,such that if the edge factor value associated with the correspondingregion in visible light image data is below a threshold value, acorresponding region in the combined image data includes no modifiedvisible light image data.
 9. The system of claim 1, wherein combiningthe modified visible light image data with the corresponding receivedinfrared image data comprises, for each of the plurality of regions ofthe modified visible light image data, blending the modified visiblelight image data with the corresponding infrared image data, theblending being performed based on a blending ratio α associated witheach of the plurality of regions, and wherein, for each of the pluralityof regions, the blending ratio α for the region is inversely related toa magnitude of the infrared image data for the corresponding region. 10.The system of claim 1, wherein the combining, for each of the pluralityof regions of the modified visible light image data, an edge midscalevalue and the corresponding edge factor value does not change thechrominance of the edge midscale value, such that each of the pluralityof regions in the modified visible light image includes the chrominanceof the edge midscale value and a luminance that is based on theluminance of the edge midscale value and the corresponding edge factorvalue.
 11. The system of claim 1, wherein each of the plurality ofregions in the visible light image data, the infrared image data, andthe combined image data correspond to single pixels.
 12. Anon-transitory computer readable medium comprising instructionsexecutable by a programmable processor that, when executed, cause theprogrammable processor to: receive visible light image data comprising aplurality of regions; receive infrared image data comprising a pluralityof regions, each of the plurality of regions of the received infraredimage data corresponding to one of the plurality of regions of thereceived visible light image data; determine, for each of the pluralityof regions in the received visible light image data, a correspondingedge factor value associated with edges in the visible light image data;generate modified visible light image data having a plurality of regionscorresponding to the plurality of regions in the visible light imagedata by combining, for each of the plurality of regions of the modifiedvisible light image data, an edge midscale value and the correspondingedge factor value, the edge midscale value corresponding to a specificcolor; and blend the generated modified visible light image data and thereceived infrared image data to create a blended image, the blendedimage having plurality of regions corresponding to the plurality ofregions in the visible light image data and the infrared image data;wherein for each of the plurality of regions in the blended image, anamount of contribution from each of the modified visible light imagedata and the infrared image data in the blended image is based on theedge factor value associated with the corresponding region in thevisible light image data.
 13. The non-transitory computer readablemedium of claim 12, wherein, for each of the plurality of regions, thecontribution of the infrared image data in the blended image variesinversely with the edge factor value associated with the correspondingregion in the visible light image data.
 14. The non-transitory computerreadable medium of claim 12, wherein, if the edge factor valueassociated with a given region in the visible light image data is belowa predetermined threshold, the corresponding region in the blended imageincludes a minimum contribution of the modified visible light imagedata.
 15. The non-transitory computer readable medium of claim 14,wherein the minimum contribution of the modified visible light imagedata in the blended image is zero percent, such that if the edge factorvalue associated with a given region in the visible light image data isbelow a predetermined threshold, the corresponding region in the blendedimage includes no modified visible light image data.
 16. Thenon-transitory computer readable medium of claim 12, wherein, for eachof the plurality of regions in the blended image, the amount ofcontribution from each of the modified visible light image data and theinfrared image data in the blended image is further based on a magnitudeof the infrared image data in the corresponding region.
 17. Thenon-transitory computer readable medium of claim 16, wherein the methodfurther comprises the step of receiving a selected edge midscale value.18. The non-transitory computer readable medium of claim 12, wherein,for each of the plurality of regions in the blended image, thecontribution from the infrared image data relative to the contributionof the modified visible light image data varies inversely with the edgefactor value.
 19. The non-transitory computer readable medium of claim12, wherein each of the plurality of regions in the visible light imagedata, the infrared image data, and the modified visible light image datacorrespond to single pixels.
 20. The non-transitory computer readablemedium of claim 12, wherein the combining, for each of the plurality ofregions of the modified visible light image data, an edge midscale valueand the corresponding edge factor value does not change the chrominanceof the edge midscale value, such that each of the plurality of regionsin the modified visible light image includes the chrominance of the edgemidscale value and a luminance that is based on the luminance of theedge midscale value and the corresponding edge factor value.